Coordinated multi‐agent hierarchical deep reinforcement learning to solve multi‐trip vehicle routing problems with soft time windows

Vehicle Routing Problem (VRP) is a widespread problem in the transportation field, which challenges intelligent level of vehicle decisions. Multi-Trip with Time Windows (MTVRPTW), as further evolved VRP considering multiple departures from one depot and temporal constraint visiting nodes, has developed into critical issues scheduling logistics, bus transit, railway, aviation. Traditionally, MTVRPTW solved by heuristic algorithm, generally time-consuming non-steady results. Reinforcement learning (RL) multi-agent framework have become popular solving to get better performance. However, lack variant dimensions searching space knowledge exchange between agents inhibit improvement algorithms. Therefore, Coordinated Multi-agent Hierarchical Deep Learning (CMA-HDRL) method proposed this study enhance overall solution quality convergence rate constructing three-layered structure (time, communication, global layers), particularly designed handle state explosion improve collaboration agents. The results show that can significantly outperform general genetic algorithm (GA), RL, hierarchical not only effectiveness on cost consisting travel time penalty but also operation robustness.